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ABSTRACT 

The  Human  Alerting  and  Interruption 
Logistics  (HAIL)  technology  is  introduced 
into  the  Aegis  surface  warfare  platform  to 
increase  warfighter  capability  to  deal  with 
high  alerting  rates.  A  spiral  development  and 
evaluation  process  investigated  the  implica¬ 
tions  of  this  new  technology  on  operator 
performance.  The  Aegis-HAIL  application,  as 
compared  to  the  B/L-7/1  alert  processing, 
increased  human  capacity  for  processing 
critical  alerts  in  four  ways:  (1)  reroute  non- 
critical  alerts  to  other  display  areas,  (2) 
provide  negotiation-based  control  to  enable 
operator-centric  navigation  of  information,  (3) 
providing  only  critical  alerts  improved  alert 
SA,  and  (4)  eased  transition  and  recovery  back 
to  the  original  task. 

INTRODUCTION 

Successful  U.S.  naval  operations  depend  on 
surface  ship  crews  taking  decisive  actions 
under  stress,  and  this  requires  good  situational 
awareness  (SA).  Alert  mechanisms  deliver  a 
significant  portion  of  this  information. 
However,  the  volume  of  alerts  has  grown 
exponentially,  overwhelming  some  operators 
and  undermining  their  ability  to  maintain  SA. 
Current  human  alerting  mechanisms  in  surface 
ship  Command  and  Control  (C2)  systems  are 
insufficient  for  future  mission  requirements. 
They  must  be  replaced  or  the  Next  Navy  will 
not  be  "fully  mission  capable." 


Human  Alerting  and  Interruption  Logistics 
(HAIL)  is  a  new  alerting  technology  being 
developed  to  improve  the  warfighter’s  ability 
to  maintain  SA  during  high  rates  of  alerting 
[McFarlane,  2002;  McFarlane  and  Latorella, 
2002].  HAIL  is  a  platform-independent,  open 
architecture  software  component  that  delivers 
human-centric  alert  mediation  support.  HAIL 
capabilities  are  realized  through  improvements 
to  the  user  interface  (UI)  that  improve  the 
operator’s  performance  in  processing  alerts 
and  through  the  reduction  in  interruptions 
during  complex,  stressful  tactical  situations. 

Lockheed  Martin  Advanced  Technology 
Laboratory  (LM  ATL)  is  leading  the  develop¬ 
ment  team  that  includes:  Lockheed  Martin 
Maritime  Systems  and  Sensors  (LM  MS2), 
Computer  Sciences  Corporation  (CSC),  Basic 
Commerce  &  Industries  (BCI),  and  the  Naval 
Research  Laboratory  (NRL). 

BACKGROUND 

Automated  notification  systems  can  perform 
the  constant  monitoring  required  to  generate 
alerts,  but  these  alerts  often  interrupt  other 
activities.  Research  has  shown  that  people  do 
not  perform  sustained,  simultaneous,  multi¬ 
channel  sampling  well;  however,  they  have 
great  capacity  to  manage  concurrent  activities 
when  given  specific  kinds  of  interface  support 
[McFarlane  and  Latorella  2002].  An  alert- 
based  information  stream  can  deliver  tasks  and 
information  that  have  the  potential  to  help 
operators  perform  effectively.  This  informa- 
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tion  source  can  support  the  capability  to  (a) 
constantly  monitor  their  dynamically  changing 
information  environments,  (b)  collaborate  and 
communicate  with  other  people  in  the  system, 
and  (c)  supervise  background  autonomous 
services. 

People  have  a  cognitive  limitation  in  dealing 
with  alert -based  interruptions.  Without  support 
for  coordinating  interruptions,  they  fail  at 
Situational  Awareness  (SA)  and  their  decision 
quality  drops.  The  Identification  Supervisor 
(IDS)  operator  for  the  Aegis  Weapon  System 
is  a  good  example.  The  IDS  is  responsible  for 
determining  and  maintaining  the  accuracy  of 
the  identity  of  hundreds  of  contacts  (aircraft 
and  other  vehicular  tracks)  visible  to  the  ship’s 
radar.  The  frequency  of  alerts  was  determined 
during  the  ASCIET  (All  Services  Combat 
Identification  Evaluation  Team  1996).  The 
IDS  operators  received  alerts  at  an  average 
sustained  frequency  of  one  every  11.5 


seconds.  The  overall  alert  frequency  for  IDS 
has  decreased  since  1996,  but  there  has  been 
net  increase  in  the  workload  for  alert-based 
information  and  tasks.  Aegis  is  currently 
“fully  mission  capable,”  however  it  will  not 
continue  to  be  without  improved  alerting 
technology. 

The  IDS  submode  is  a  primary  candidate  for 
HATE  technology.  HAIL  will:  (1)  reduce  the 
number  of  interruptions,  (2)  improve  the 
operators’  SA  for  each  alert,  (3)  improve 
control  of  alerts  requiring  action  responses,  (4) 
improve  SA  for  element  status  information, 
and  (5)  assist  in  returning  to  the  operator’s 
original  task  [NSWCDD  2003].  Figure  1 
depicts  a  comparison  of  today’s  alert 
processing  versus  using  the  HAIL  technology. 
Figure  2  shows  the  specific  processing  path 
that  the  current  Aegis  operator  would  take  to 
process  alerts. 
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FIGURE  1.  The  Current  Aegis  Alerting  System  versus  the  HAIL- Aegis  Alerting  System. 
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FIGURE  2.  The  Aegis  BL7P1  IDS  console  showing  the  steps  required  to  process  alerts. 


APPROACH 

The  Taxonomy  of  Human  Interruption 
[McFarlane  and  Latorella  2002]  identifies  four 
methods  for  coordinating  the  interruption  of 
people:  (1)  immediate,  interrupt  them  now  and 
get  it  over  with,  (2)  negotiated,  give  the 
recipient  the  authority  to  negotiate  when,  if, 
and  how  they  will  handle  interruptions,  (3) 
mediated,  a  third  party  brokers  all  interrup¬ 
tions  and  decides  when  and  how  best  to 
interrupt  the  recipient,  and  (4)  scheduled,  a 
prearranged  convention  for  the  times  to 
interrupt  the  user. 

Empirical  research  with  human  subjects  found 
that  negotiation-based  interruptions  best 
support  operator  performance  except  where 
small  differences  in  the  timeliness  of  handling 
interruptions  is  critical;  then  the  immediate 
solution  is  best  [McFarlane  2002],  The  HAIL 
technology  is  founded  on  this  result  and  is 
engineered  to  provide  negotiation-based 
coordination  support  to  increase  naval 
operators’  capability  to  work  effectively 
during  high  rates  of  alerting. 


The  HAIL  negotiation-based  coordination 
support  provides  the  capability  to:  (1)  quickly 
notice  each  alert  and  the  precise  minimum 
information  needed  to  make  instant  judgments 
of  alert  criticality,  (2)  negotiate  control  over 
when  and  how  to  process  alerts,  (3)  easily 
visualize  the  contents  of  alert-based  informa¬ 
tion  without  explicit  user  interface  actions 
required  of  operators,  (4)  differentiate  among 
alerts  based  on  context  and  processing 
requirements,  and  (5)  automatically  help 
resume  pre-interruption  work  after  completing 
an  alert. 

METHOD 

The  first  objective  of  the  HAIL  team  was  to 
demonstrate  the  technology  maturity  and 
operational  utility  of  HAIL  at  TRL7  in  May 
2004 '.  The  team’s  second  objective  was  to 


1  The  Technology  Readiness  Levels  (TRL)  range  from 
values  of  1  -  9  where  TRL1  =  “Basic  principles 
observed  and  reported”  to  TRL9  =  “Actual  system 
proven  through  successful  mission  operations.”  TRL7 
=  “System  prototype  demonstration  in  an  operational 
environment.” 


facilitate  the  transition  of  HAIL  into  the  Aegis 
Open  Architecture  platform.  Figure  3  depicts 
the  nine  TRL  levels  and  Figure  4  depicts  were 
these  levels  where  satisfied  within  the  HAIL- 
SS  program  plan. 


have  been  used  in  8  full  cycles  of  redesign  and 
testing.  The  final  solution  is  based  on  this 
foundation  and  has  been  proven  useful  under 
highly  realistic  conditions. 

SUMMATIVE 
EVALUATION 


Participants 

Twenty-two 
experienced  naval 
operators  participated 
from  the  Aegis 
Training  and 
Readiness  Center 


FIGURE  3.  Technology  Readiness  Levels 
Rating  Scale  for  HAIL  Technology. 


FIGURE  4.  HAIL  Technology  Reached  TRL7 
through  a  Spiral  Development  Program  Plan. 


HAIL  applied  a  human-centered  analysis  and 
design  process  to  define,  develop,  and  test  the 
technology.  The  analysis  includes  two  initia¬ 
tives:  (1)  evaluation  of  20  years  of  Aegis 
‘lessons  learned’  data  related  to  alert 
processing  and  (2)  user  experience  efforts  that 
include  current  fleet  sailors  to  assess  current 
alert  processing.  These  efforts  have  enabled 
the  HAIL  team  to  develop  a  prototype 
interface  that  has  been  submitted  to  five  cycles 
of  formal  usability  testing  and  two  cycles  of 
formal  evaluation.  Each  of  these  tests  clarified 
the  strengths  and  weaknesses  of  the  Aegis 
HAIL  user  interface  (UI)  and  these  results 


(ATRC)  Norfolk  and 

from  ATRC  Dahlgren  (five  were  former  Navy 
personnel  and  now  contractors).  They  had 
combined  experience  of  over  280  years  (3-28 
years  experience  individually,  14.5  years  on 
average).  Technical  problems  with  data 
collection  prevented  the  use  of  the  data  from 
two  participants. 

Evaluation  Design  and  Conditions 

A  “within  subjects  single  factor  Latin 
Squares”  design  was  used.  The  testing 
evaluated  two  conditions:  a  control  condition 
without  the  HAIL  engine,  Baseline  7  Phase  1 
(BL7P1),  and  an  experimental  condition  with 
the  HAIL  engine,  Aegis  HAIL.  Each  subject 
was  tested  in  both  conditions.  The  order  in 
which  they  were  tested  was  counterbalanced, 
and  subjects  were  randomly  assigned  to  the 
two  orders.  Each  condition  was  tested  using  a 
different  scenario,  scenario  1  or  2.  Ten  of  the 
twenty  participants  were  given  the  BL7P1 
condition  first,  five  with  scenario  1  and  the 
other  five  with  scenario  2.  Ten  of  the  twenty 
participants  were  given  the  HAIL  condition 
first,  five  with  scenario  1 ,  the  remaining  five 
with  scenario  2. 

Task 

The  highly  realistic  scenario  was  designed  and 
developed  by  former  Aegis  combat  officers. 


The  summative  evaluation  used  two  similar 
35-minute  scenarios  to  allow  for  cross-subject 
and  intra-subject  comparison.  The  scenarios 
were  composed  mostly  of  action  alerts  (70%), 
with  a  minimal  amount  of  informational  alerts 
(30%).  The  scenarios  were  designed  and 
scripted  using  ACTS  Exercise  Request 
Development  System  II  (AERDS  II)  software 
to  simulate  a  Link  16  tactical  picture  of  the 
Arabian  Gulf  with  240-260  tracks.  The 
AERDS  files  were  then  converted  to  a 
character-delimited  file  capable  of  being 
handled  by  the  Altia  mockup.  Each  scenario 
contained  a  5-minute  warm  up  period 
followed  by  a  15-minute  low-stress  period  and 
a  15-minute  high-stress  period  of  action. 
Within  the  high-stress  period,  there  is  a  10-15 
second  surge  of  alerts  where  the  test 
participant  is  bombarded  with  an  alert  every 
second  for  the  duration  of  the  surge. 

Apparatus 

The  formative  evaluation  was  conducted  on 
Aegis  BL7P1  AN/UYQ-70  console  suites 
running  in  the  Aegis  CSETC  testing  lab  at  LM 
MS2,  Moorestown,  NJ.  Evaluating  on  the 
actual  Aegis  hardware  dramatically  increased 
the  external  validity  of  our  findings.  However, 
the  actual  Aegis  hardware  did  not  adequately 
support  the  type  of  simulation  capabilities 
necessary  for  rigorous  evaluation.  To  meet  our 
internal  validity  requirement,  the  evaluation 
platform  needed  the  following  attributes  that 
could  not  be  achieved  in  an  Aegis  hardware 
platform:  (1)  Able  to  simulate  a  large  number 
of  tracks  and  alerts  without  overloading 
system  capabilities;  (2)  Scenarios  are  repeat 
able  in  the  exact  configuration  of  tracks  and 
alerts  for  numerous  runs  with  multiple 
subjects;  (3)  All  user  actions  that  need  to  be 
logged  can  be  reliably  captured  and  time- 
stamped  (preferably  with  automated  capture 
methods);  (4)  Scenario  and  environment  are 


extremely  realistic  replications  of  the 
operational  environment;  and  (5)  Evaluation 
efforts  remain  cost-effective  and  reasonable  to 
complete  with  available  resources. 

An  evaluation  platform  was  created  to  satisfy 
these  evaluation  requirements.  This  platform 
is  a  hybrid  combination  of  the  real  HAIL 
system  running  on  a  Solaris  workstation  fully 
integrated  with  a  mockup  of  the  rest  of  the 
Aegis  displays  built  with  Altia  Design.  The 
result  is  an  extremely  realistic,  fully  functional 
replication  of  the  BL7P1  system  with  the 
scenarios  being  driven  by  a  local  simulator. 
Integrating  HAIL  into  this  platform  enabled  a 
rigorous  engineering  assessment  of  the  HATE 
code  and  user  evaluation  data.  The  resulting 
platform  was  designed  to  present  stressful 
scenarios  that  exercised  HAIL  capabilities  and 
to  log  all  user  actions  for  analysis  purposes. 

Procedure 

Evaluation  data  were  recorded  with  three 
objective  measures  and  five  subjective 
instruments.  Objective  data  were  recorded 
with:  (1)  automatic  logging  of  all  user 
interface  events;  (2)  two-channel  synchronous 
digital  video  of  the  operators  interaction  with 
the  console;  and  (3)  expert  observers  to  assess 
operator  performance  that  could  not  be 
automatically  logged.  Subjective  data  were 
recorded  with  four  questionnaires  and  an  exit 
interview.  The  four  questionnaires  used  were: 
(1)  the  NASA  Task  Load  (TLX)  to  measure 
workload;  (2)  a  modified  NASA  Bipolar 
Rating  Scale  to  measure  stress  level;  (3)  a 
HAIL  functionality  scale  to  measure  utility  of 
the  new  functions;  (4)  a  comparison  of  HATE 
to  the  Aegis  BL7P1  baseline  without  HAIL  to 
measure  the  relative  value  of  the  before  and 
after  solutions;  and  (5)  formal  exit  interview 
recorded  with  a  digital  audio  recorder. 


Results 

The  two  main  utility  assessment  objectives  for 
the  HAIL-SS  Phase  2  Evaluation  mentioned  in 
Section  1.2,  Evaluation  Goals:  (1)  Which 
HAIL  capabilities  are  most  valuable  for 
improving  warfighter  performance?  and  (2) 
Does  the  introduction  of  HAIL  cause  any 
degradation  in  warfighter  performance;  i.e., 
“did  we  break  anything?”  can  be  answered  by 
looking  at  the  results.  HAIL  was  found  to 
increase  warfighter  performance  in  four  ways: 
immunity  to  the  effects  of  trash  alerts,  fewer 
interruptions,  better  alert  SA,  and  easier 
recovery  of  non-alert  work  after  handling 
alerts.  Through  discussions  with  actual 
operators,  it  has  been  determined  that  errors 
can  cause  meaningless  noise  alerts  to  be 
inserted  into  the  queue  along  with  important 
alerts.  These  important  alerts  become  buried  in 
the  alert  queue  beneath  the  noise  alerts.  HATE 
makes  it  easy  for  operators  to  see  the 
difference  between  noise  alerts  and  important 
alerts.  Operators  can  sort  through  and  select 
the  important  alerts,  eliminating  the  critical 
problem  that  the  noise  alerts  can  present. 

Results  show  that  HAIL  produces  a  reduction 
in  interruption  frequency  by  sending  fewer 
alerts  to  the  operator.  Redirecting  status  alerts 
to  the  Status  Display  Window  reduces  the 
number  of  alerts  presented  to  the  operator  in 
the  alert  queue.  While  the  SAGAT  scores  of 
the  Baseline  Aegis  and  the  HAIL  systems 
showed  no  statistical  difference  in  the 
Situational  Awareness  (SA)  score,  the 
Difference  questionnaire  reported  a  higher  SA 
in  the  HAIL  condition.  Participants  perceived 
that  their  overall  Situation  Awareness  was 
better  with  HAIL  than  without  it. 

HAIL  increases  operator  capability  to 
efficiently  switch  between  alert-based  work 
and  non-alert  work.  HAIL  automatically  resets 
the  TACSIT  configuration  to  its  pre-alert 


settings  (e.g.  range  scale)  and  therefore 
reduces  the  amount  of  time  and  effort  required 
for  the  operator  to  retrieve  his  pre-alert 
context. 

In  an  analysis  of  the  surge  phase  only,  the 
HAIL  condition  produced  an  overall  average 
reduction  in  surface  latency  that  was  greater 
than  the  Baseline.  This  result  indicates  that 
HAIL  accomplishes  one  of  the  main  goals  the 
technology  set  out  to  achieve:  it  allows 
operators  to  find,  surface,  and  address  critical 
alerts  in  a  timely  manner  during  high-stress 
periods  in  the  operational  environment. 

Subjective  data  was  analyzed  primarily  to 
determine  which  features  have  been 
successfully  designed  and  implemented  to  be 
useful  to  the  operational  community.  The  most 
important  findings  of  the  subjective  data 
include:  (1)  Advantage  ratings  of  HAIL  over 
Aegis  baseline: 

a.  Ability  to  recognize  important  alerts 

b.  Ability  to  surface  important  alerts 

c.  Minimize  less  important  alert  work 

d.  Alert  Situation  Assessment 

e.  Alert  tool  control 

Ligure  5  shows  the  average  relative  rating  for 
HAIL  capabilities  versus  Aegis  BL7P1 
capabilities  without  HAIL.  The  results  are 
sorted  from  highest  rated  HAIL  capabilities  to 
least. 

HFE  IMPLICATIONS 

Operator  Involvement  in  Scheduling  Alerts 

In  the  current  Aegis  alert  design  operators 
must  scroll  through  each  alert  to  see  the  full 
alert  message.  The  HAIL  approach  allows  the 
operator  to  review  all  alerts  and  select  the  ones 
that  require  immediate  attention.  In  addition, 
there  are  visual  indications  of  the  associated 
track  and  pertinent  positional  information  to 
help  determine  whether  or  not  to  address  it 
immediately. 


Advantage  ratings:  Aegis  HAIL  vs.  Aegis  baseline  (1st  of  4) 
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Advantage  ratings:  Aegis  HAIL  vs.  Aegis  baseline  (2nd  of  4) 
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Advantage  ratings:  Aegis  HAIL  vs.  Aegis  baseline  (3rd  of  4) 
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Advantage  ratings:  Aegis  HAIL  vs.  Aegis  baseline  (4th  of  4) 
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FIGURE  5.  The  sorted  average  relative  ratings  of  HAIL 
capabilities  compared  to  non-HAIL  Aegis  BL7P1 
capabilities. 


Operator  Responsibility  for 
Judging  Alert  Criticality 

While  the  combat  system 
displays  the  alerts  in  the  order 
of  receipt,  the  operator  is  able  to 
view  all  the  alerts  in  the  queue 
and  assess  the  relative  criticality 
of  one  alert  versus  another.  If 
the  operator  is  responsible  for 
pre-engagement  and 
engagement  response  they 
obviously  will  respond  to  these 
alerts  before  others. 

Improved  Visualization  of 

Alert-Generated 

Information 

The  HAIL  interface  has  been 
designed  to  visually  distinguish 
between  the  types  of  alerts. 

There  are  ‘action’  and  ‘inform- 
tion’  alerts  in  the  Aegis  system. 
While  the  ‘action’  alerts  are 
listed  in  the  new  alert  queue,  the 
‘information’  alerts  relative  to 
equipment  status  are  rerouted  to 
the  “Status  Board”  display. 

Mediation  Control  Suite 

Unlike  the  Aegis  system  that 
only  provides  a  single  button  for 
processing  alerts,  the  HAIL 
system  provides  five  buttons 
that  allow  the  operator  to  better 
manage  the  alerts. 

Improved  Context 
Switching 

Currently,  the  operators  have  to 
find  their  own  way  back  to  the 
pre-alert  task.  HAIL — when  the 
operator  indicates  that  the  alert 
has  been  ‘completed’ — returns 


the  original  displays  and  controls  to  the 
tactical  display  surface. 

Improved  Situational  Awareness 
between  the  Graphical  TACSIT  and  the 
Textual  Alert  List 

The  HAIL  system  provides  operators  with  an 
enhanced  level  of  situational  awareness. 
Currently,  the  operator  must  surface  the  alert 
in  order  to  see  the  associated  track  on  the 
TACSIT  display.  With  the  HAIL  interface,  the 
operator  mouses  over  the  alert  in  the  list  to  see 
the  highlighted  track  on  the  TACSIT.  This 
visually  confirms  the  track’s  position  and  aids 
in  tactical  decision-making. 

CONCLUSIONS 

The  most  significant  impact  of  HAIL 
technology  into  Aegis  is  the  increase  in  the 
operator’s  capability  to  remain  in  constant 
contact  with  the  stream  of  critical  alert-based 
tasks  and  information  regardless  of  the 
alerting  rate  or  the  amount  of  non-critical 
alerts.  Operators  are  able  to  recognize  critical 
alerts  as  they  arrive  and  can  then  reach  into  the 
alert  queue  and  pull  out  whatever  is  most 
important  at  the  moment. 
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